Who gets grant money? The (gendered) words decide.

Blinded reviews for scientific research grants — in which all identifying information on the applicant is removed — are designed to remove gendered outcomes, among other biases.

They may not be working.

Female scientists are 16% less likely than men to get a high score on their grant proposal, and new research suggests word choice might be the reason why.

“Despite blinded review, female applicants receive significantly lower scores,” write MIT Sloan professor Fiona Murray and co-authors Julian Kolev, a Southern Methodist University assistant professor, and Yuly Fuentes-Medel, former MIT Sloan post-doctoral fellow and current project manager for Fiber Technologies. “We find strong gender differences in the usage of broad and narrow words, suggesting that differing communication styles are a key driver of the gender score gap.

The findings are reported in a new working paper, “Is blinded review enough? How gender outcomes arise even under anonymous evaluation."

x x

According to the researchers, women used “narrower” words that are “highly concentrated in a small number of topics.” Male applicants, on the other hand, tended to use “broad” words, which “appear at similar rates in all topic areas,” the study stated.

Share

Women in a blinded review of grant proposals had a 16% lower chance of scoring highly than men did.

Women’s use of topic-specific words lowers their chance of getting a high score from reviewers compared to men.

"Even after accounting for the quantity and quality of male and female applicants, there is still a disparity in outcomes that favors men over women,” Kolev said. “This is a problem not only because of the general issue of underrepresentation of women in science and technology, but also because this disparity strongly suggests that the reviewers are allocating funding inefficiently and failing to make the best use of their organization’s resources."

The researchers gathered their data from nearly 6,800 grant proposals submitted by U.S.-based applicants to the Bill and Melinda Gates Foundation between 2008 and 2017. Specifically, the proposals were for the Global Challenges: Exploration Program, which bases its funding decisions on anonymous proposals independently reviewed by a diverse pool of reviewers.

To understand differences among the proposals, the researchers took the top 1,000 most frequently used words in the grant proposal dataset. This group included words like new, HIV, using, and cell.

While the total number of frequently used words is roughly the same for men and women — and women did use some high-scoring words like transmission, human, and model — "female applicants used fewer of the words favored by reviewers when describing their proposals, and more of the words associated with low reviewer scores,” the co-authors write.

Words used by men and found in proposals that scored highly include terms like detection, bacteria, and drug. Meanwhile, words that were used often by women and found in low-scoring proposals include ones like vaccination, children, and health.

When these narrow and broad words are plotted and compared to high- and low-scoring proposals, “the overall pattern suggests that words falling well below the 45-degree line (i.e. those with strongly negative score disparities) are more likely to be used disproportionately by female applicants,” the researchers said. “By contrast, most ‘male’ words are likely to be near or above the 45-degree line, indicating a positive score disparity.”

The result, the researchers write, is a “significant reduction” in how reviewers perceive the quality of the women’s proposals. And that’s even when those proposals are of high quality.

In fact, the researchers found that women who received grants during the time of the study “may well generate a greater ‘return’ on resources compared to their male counterparts."

For example, female applicants generated 37% more new grants from the National Institutes of Health after receiving funding from the Gates Foundation.

“While obtaining funding from the NIH or similar organizations is not an end-goal by itself, this additional funding can be reasonably expected to serve as an input to future research and innovation,” the researchers write.

The researchers also found that in every outcome they measured, the impact of being or not being funded is larger for female applicants. Combined with the NIH grant funding results, the researchers write, “this suggests that while non-funded women perform worse than non-funded men by as much as 26% depending on the outcome we track, the gender gap goes away when you look at funded applicants.”

In other words, the researchers write, the Gates foundation funding helps “level the playing field between male and female researchers.”

Other disparities the researchers found in their study include:

• Female applicants made up only one-third of the applicant pool.

• Female applicants were roughly 30% less senior than their male counterparts.

• Female applicants “are significantly less likely” to reapply after an initial rejection.

Making improvements at an organizational level

So what can female applicants do to increase their chances for a grant? Unfortunately, there’s no easy answer. 

Murray said the burden lies equally with funders and applicants. Kolev said while female applicants could alter their communication style in pursuit of grant funding, doing so carries the potential risk of undermining the quality of their research.

“We believe that the best opportunities for improvement are at the level of organizations and reviewers,” Kolev said. “Most importantly, organizations could implement reviewer training to limit their sensitivity to communication style, or modify applications to focus on technical descriptions and limit the use of broader language that [in our findings] is favored by reviewers but fails to correlate with follow-on success for the proposal.”

Kolev and his co-authors suggested an organization could also increase the number of female reviewers, as the study showed women don’t favor proposals from men at the same rate as male reviewers favor male applicants.

The researchers said they don’t recommend doing away with blinded reviews, as that type of judging has shown overall benefits in equalizing opportunities for candidates across a variety of categories like gender and race.

However, if there is one thing for female applicants to take from their research, the co-authors write that repeat applicants received higher scores than their first proposals, suggesting “the value of persistence in the face of rejection.”